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Abstract 

We study the orbit behavior of a four dimensional smooth symplec¬ 
tic diffeomorphism / near a homoclinic orbit T to an 1-elliptic fixed 
point under some natural generic!ty assumptions. 1-elliptic fixed point 
has two real eigenvalues out unit circle and two on the unit circle. Thus 
there is a smooth 2-dimensional center manifold where the restric¬ 
tion of the diffeomorphism has the elliptic fixed point supposed to be 
generic (no strong resonances and first Birkhoff coefficient is nonzero). 
Moser’s theorem guarantees the existence of a positive measure set of 
KAM invariant curves. itself is a normally hyperbolic manifold in 
the whole phase space and due to Fenichel results every point on 1U° 
has 1-dimensional stable and unstable smooth invariant curves form¬ 
ing two smooth foliations. In particular, each KAM invariant curve 
has stable and unstable smooth 2-dimensiona invariant manifolds be¬ 
ing Lagrangian. The related stable and unstable manifolds of are 
3-dimensional smooth manifolds which are assumed to be transverse 
along homoclinic orbit T. One of our theorems present conditions un¬ 
der which each KAM invariant curve on in a sufficiently small 
neighborhood of T has four transverse homoclinic orbits. Another re¬ 
sult ensures that under some Birkhoff genericity assumption for the 
restriction of / on lU'’ saddle periodic orbits in resonance zone also 
have homoclinic orbits though its transversality or tangency cannot be 
verified directly. 


1 Introduction and set-up 

Any tools that can help to understand, if a given Hamiltonian system is 
integrable or non-integrable and therefore has a complicated orbit behavior, 
are of the great importance. There are well known criteria based on the 
Melnikov method |33l [26l |29l |371 [TO], bnt they are mainly applicable for 
systems being nearly integrable. 
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There exists other class of criteria based on the study of the orbit behav¬ 
ior in definitely non-integrable systems: if we know that some structures in 
the phase space are met only in non-integrable systems, then we may take 
the existence of such a structure in the phase space of a system under consid¬ 
eration as a criterion of its non-integrability. Such criteria are most efficient, 
if the structures mentioned can be rather easily identihed. To this type of 
criteria one can refer those based on the existence of homoclinic orbits to the 
different type of invariant sets, the most popular are those related with ho¬ 
moclinic orbits to different types of equilibria, periodic orbits and invariant 
tori ISlIiaESlEolEalElEaillH]. Surely, non-integrability criteria are not 
the unique goal of the study: a much more interesting and hard problem is 
to describe possible types of the orbit behavior in the system near such the 
structure and its changes when parameters of the system vary. 

In the paper we study a C^-smooth, r > 6 , symplectic diffeomorphism / 
on a C°°-smooth 4-dimensional symplectic manifold (M, n), 14 is C°°-smooth 
non-degenerate 2-form. We assume / to have an l-elliptic hxed point p, that 
is, differential Dfp has one pair of multipliers on the unit circle and a 
pair of real multipliers p ^ ±1. Below we suppose p to be positive 

and 0 < p < 1. We call such fixed point to be an orientable l-elliptic point. 
The fixed point with negative p, we call to be non-orientable. The non- 
orientable point becomes orientable, if one considers instead of /. 

Near an l-elliptic fixed point there is a C^“^-smooth 2-dimensional in¬ 
variant symplectic center submanifold corresponding to multipliers 
(SHIS]. The restriction of / on is a C”' ^-smooth 2-dimensional sym¬ 
plectic diffeomorphism and p is its elliptic fixed point. We assume p to be 

of the generic elliptic type | 2 , that is, strong resonances are absent in the 
TT 27r 

system (a ^ —, —) and the first coefficient in the Birkhoff normal form 
2 3 

for f\^rc does not vanish. In this case we shall call an l-elliptic hxed point 
to be a generic l-elliptic fixed point. Then the Moser theorem |36j is valid 
for the restriction f\^yc near p, this gives a positive measure Cantor set of 
closed invariant curves on which enclose p and are accumulated to it. 
The needed minimal smoothness for a symplectic diffeomorphism is 5 due to 
[38]. This explains the inequality r > 6 . 

Center manifold is a normally hyperbolic invariant manifold in the 
sense of dSlEI] and has its local ^-smooth 3-dimensional stable manifold 
local C^^^-smooth 3-dimensional unstable one since two other 
multipliers p, are lesser than 1 and greater than 1 , respectively (these 
two local 3-dimensional manifolds for the hxed point p are simultaneously 
center-stable and center-unstable manifolds, respectively, this explains our 
notations). These manifolds can be extended till the global ones by the 
action of f~^ and /, respectively. The extended manifolds will be denoted 
as and 

Each invariant KAM-curve 7 on can be considered as being saddle 
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one, since it has local 2 -dimensional stable and unstable manifolds which can 
be also extended till global manifolds 14^^( 7 ), VK“( 7 ) by the action of 
Topologically these manifolds are local cylinders, both being Lagrangian 
submanifolds in M [T]. The existence and smoothness of these manifolds 
relies on the results of [13 ES] and will be proved in Appendix. 

Fixed point p has also two C’'-smooth local invariant curves through p 
being its local stable Wf^^{p) and unstable Wl^^{p) manifolds |22| . Their 
extensions by the action of f~^ and / are C^-smooth invariant curves W‘^{p) 
and W'^{p), respectively. 

Our first two assumptions in the paper concern the existence of a homo¬ 
clinic orbit to p and its type. 

Assumption 1 (Homoclinic intersection) Curves W'^{p) andW^{p) ha¬ 
ve an intersection at some point q, thus generating a homoclinic orbit T to 
fixed point p. 

Assumption 2 (Transversality condition) Manifolds W^{p) andW^'^{p) 
are transverse at point q and, hence, along T. 

Later on in the section we will construct linear symplectic scattering 
map S which acts on tangent plane TpW‘^ and describes in the linear approx¬ 
imation an asymptotic behavior of orbits close to T after one-round travel 
near T. The restriction of differential Dfp on symplectic invariant plane 
TpW^ C TpM is a linear symplectic 2-dimensional map with two eigenval¬ 
ues and, therefore, this plane is foliated into closed invariant curves 

of the map. Every such a curve is an ellipse, all of them can be obtained 
from the one multiplying their vectors at positive constants. Fix one such 
ellipse E C TpW^. Then its image S{E) is also an ellipse (usually not from 
the foliation) with the same center at the origin and of the same area with 
respect to the restriction of 2-form n on this plane. Thus, the intersection 
E n S{E) consists of either four points (a generic case) or these two ellipses 
coincide (a degenerate case). In the first case the intersection of two ellipses 
is transverse at every of four points. 

Assumption 3 (Genericity condition) The intersection EnS{E) is trans¬ 
verse and therefore consists of four points. 

It is evident that this assumption does not depend on the explicit choice of 
the ellipse E. This condition allows one to select a generic case and provides 
the mean to verify this. 

Our first result is the following theorem. 

Theorem 1 Let a f-dimensional symplectic diffeomorphism f with 1-elliptic 
fixed point p obeys Assumptions 0110 Then there is a sufficiently small 
neighborhood U of homoclinic orbit T such that every closed invariant KAM- 
curve on W’^{p) H U possesses four transverse homoclinic orbits in U. 


3 


Intersection of invariant manifolds of the diffeomorphism / in the neigh¬ 
borhood of homoclinic orbit are sketchy represented on Fig. It is worth 
remarking that for our case center manifold as was mentioned, is nor¬ 
mally hyperbolic two-dimensional invariant manifold on which the restriction 
of / is a twist map. Thus our results on existence of transverse homoclinic 
orbits to invariant KAM curves are connected with the study of Hamiltonian 
dynamics near low-dimensional invariant whiskered tori initiated in m and 
extended in many recent papers (see, for instance, reviews mm 

Before going to the proof, let us recall some related results for Hamilto¬ 
nian vector helds i3oi Ea EH Ea CHI [m [35]. Homoclinic orbits to a saddle- 
center equilibrium for a real analytic Hamiltonian system with two degrees 
of freedom, namely, for restricted circular three body problem, were found 
numerically in m and proved to exist analytically through asymptotic ex¬ 
pansions in • The problem on the orbit behavior of a real analytic Hamil¬ 
tonian system near a homoclinic orbit to a saddle-center equilibrium was hrst 
set up and partially solved in IMl, though it was earlier discussed in |7|- In 
particular, the existence of four transverse homoclinic orbits to every small 
(Lyapunov’s) periodic orbit on the center manifold of the saddle-center was 
proved in m using the Moser normal form and the genericity condition was 
found hrst in |30) . In |18| under an additional assumption that a homoclinic 
orbit to a saddle-center belongs to some invariant symplectic 2-dimensional 
submanifold (that is generically not the case), the genericity condition was 
reformulated in terms of the related scattering problem for the transverse 
2-dimensional system linearized at the homoclinic orbit. It was hrst discov¬ 
ered in j35| and in a more rehned invariant form in [T^ that in a generic 1- 
pararneter unfolding of reversible 2 d.o.f. Hamiltonian systems that unfolds 
a Hamiltonian system with a symmetric homoclinic orbit to a symmetric 
saddle-center equilibrium, there exists a (self-accumulated) countable set of 
parameter values near the critical one such that for a poiut of this set the 
related Hamiltouiau system has a homoclinic orbit to its symmetric saddle- 
center. Usually these latter orbits are multi-round with respect to the initial 
homoclinic orbit. Several applications, where non-integrability of a system 
under consideration was proved using this method, can be found in |6l I20j . 
A partial extension of results to the case of Hamiltonian systems with n de¬ 
grees of freedom, n > 3, having a center-saddle equilibrium (one pair of pure 
imaginary eigenvalues and the remaining ones with nonzero real parts) with 
a homoclinic orbit, was given in |24| . Here the scattering map was extended 
onto the case when the center manifold is 2-dimensional but the dimension 
of transverse directions is 2(n — 1). 

In fact, the results we discuss here refer to a 3 d.o.f. Hamiltouian system 
ou a C^^-smooth symplectic manifold with a smooth Hamiltonian H such 
that Xh has a periodic orbit C of the center-saddle type. The latter means 
the multipliers of this orbit (except for the common double unit) are a pair 
gibio ^ p^jj. q£ reals /i, ^ 7 ^ ±1. Such periodic orbit has 2-dimensioual 
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Figure 1: Invariant manifolds in the neighborhood of homoclinic orbit 
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stable and unstable invariant manifolds through C, they both belong to 5- 
dimensional level H = H(C). If these manifolds have an intersection along 
some orbit F, then this homoclinic orbit tends to C as t —)• ± 00 . Choose 
some cross-section N to the flow through a point p € C in 5-dimensional level 
H = H{C). We get a four-dimensional symplectic (w.r.t. the restriction of 
2-form O to N) local Poincare diffeomorphism f : N ^ N with fixed point p 
of the 1-elliptic type (corresponding to C) defined in a neighborhood U <Z N 
of p. Intersection of stable and unstable manifolds of C with N give smooth 
local curves through the fixed point, the traces of F in N form a countable 
set of homoclinic points accumulating at p. Fix one homoclinic point q- ^ U 
on the unstable curve and one homoclinic point € U on the stable curve. 
Choose some small neighborhoods V- G U of and V+ C C/ of on N. 
Flow orbits define a symplectic map F : V- ^ V+, F{q-) = q^, that we call 
as global one. Then a symplectic first return map defined as / for points 
which belong to U \V- and as F for points in V- is a map we discuss. 

The local center manifold VF'^(C) for periodic orbit C is of dimension four, 
it contains the symplectic cylinder filled with periodic orbits (continuations 
of C onto close levels of H) and if conditions of Theorem hold, then the 
restriction of the system on W^{C) has a positive measure set of invariant 2- 
dimensional tori with Diophantine rotation numbers. When we fix the level 
H = H{C), then its intersection with the center manifold is 3-dimensional. 
Every torus T has stable and unstable 3-dimensional manifolds which inter¬ 
sect each other along four transverse homoclinic orbits to the torus within 
5-dimensional level H = H{C). 

2 Consequences of the transversality condition 

Due to Assumptions and 1^ two smooth 3-dimensional manifolds W^^{p) 
and W^“(p) intersect transversally at a homoclinic point q and thus along 
a smooth 2-dimensional disk S containing q. This disk is symplectic w.r.t. 
2-form u! being the restriction of 2-form O = dx A dy + du A dv on S. Indeed, 
in section]^ it will be proved that in normalized coordinates, in which 0 = 
dx A dy + du A dv, disk (more exactly, some its finite iteration under /) will 
have the following representation: 

X = ‘h(tt, v), y = 0. 

This implies S be symplectic w.r.t. 2-form uj = du A dv. The following 
lemma is valid: 

Lemma 1 If Assumptions ^and^ hold, then S and W‘^{p) are transverse 
at q within lT™(p) and, therefore, W^{p) and VF“(p) are also transverse at 

q- 
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Proof. To prove this lemma, we use some symplectic coordinates (x, u, y, v), 
Q = dx A dy + du A dv, in a neighborhood V of point p in which manifolds 
and are straightened, that is they are given as x = 0 (for 1 T'^“) 
and y = 0 (for 1T“). In addition, in these coordinates local stable manifold 
W^{p) is given asy = u = v = 0 and local unstable manifold W'^{p) is done 
as X = n = n = 0. The existence of such coordinates is proved in Appendix. 
We also assume that q € V. Since orbit T through q is homoclinic, then 
there is an integer > 0 such that f~'^{q) € V for all n > N. Denote qi 
the point f~^{q) G V and let P be the tangent to lT“(p) at qi. Denote 
L = Df^ : Tqj^M —)• TqM, then T(/“) is transversal to TqlT“ in virtue to 
Assumption(transversality condition). Set D = L(rgjlT™) Pi TglT^, D is 
2-dimensional plane. One needs to prove that (the tangent to W^{p) at 
q) does not belong to D, that is intersects D at only one point. For linear 
symplectic map L the following matrix representation holds: 


L = 


/on 

012 

bn 

^12\ 

021 

O 22 

b2i 

b22 

Cll 

C 12 

dll 

di2 

\C21 

C 22 

d2i 

d22/ 


where a, b, c, d are 2 x 2 -matrices. Since 1T“ are straightened in coor¬ 
dinates we use, tangent spaces to VF™, W'^{p) at qi and tangent spaces to 
VF“, W^{p) at q are written as follows: 


/0\ 


/x\ 




/x\ 

u 

, = 

u 

, r = 

0 

, 1" = 

0 

y 

■ y 

0 


y 


0 

\v/ 


\v) 


\o) 


VO/ 


Transversality of L(l^) and TqW^ is expressed as du 7 ^ 0 in matrix L. 
Indeed, one has L{1^) = (&iiy, 6212 /, dny, ^ 212 /)^ (vector-column). Transver- 
sality of T(/“) and TqW‘^^ means that determinant 


bu 

b2i 

dll 

d2i 


1 0 
0 1 
0 0 
0 0 


= d 
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does not vanish. 

The plane D is given by the set of solutions of the system Q: 

'X = ai2U + buy -|- b^v 

JJ . ^ ^ = + b2iy + b22V 

■ I 0 = Ci 2 n + duy + di 2 V 
= C 22 U + d2iy + d22V 


( 1 ) 
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If P C -D then in the system above u = 0, h = 0 for all {u,v). Expressing 
y from the third equation in Q and inserting into other equations we get 
a parametric representation of plane D (with parameters u,v). Consider 
separately subsystem ([^: 


022 ^ + b 2 iy + h 22 V = u 

C 22 U + d2iy + d22V = V. 


( 2 ) 


Due to inequality du 7 ^ 0 we can express y = {—C 12 U — di 2 v)/dii from Q 
and insert it into ([^: 


f diia 22 U + h 2 l{-Ci 2 U - di 2 v) + diib 22 V = u 
\d 11 C 22 U + d2l{-Ci2U - di2v) + diid22V = V. 


(3) 


Let us calculate the determinant of the system (|^. To this end, we rewrite 
it in the following form: 

u{dlia22 - b2lCl2) + v{b22dii - b2ldl2) = u 
u{diiC22 — d2lC\2) + v{d22dll — d2ldi2) = V. 

This determinant is calculated as follows: 

A = (dlia22 — b2lCl2){d22dll — ^21^12) — (&22dll — &2ldl2)(dllC22 — d2lCl2) 
= ^^^022^22 — hiia22d21<^12 “ ^21012^22^^11 + fe2lCl2d2lC^12 “ dnC22&22 
+ ^ 22 ^ 11^21012 + & 2 lhl 2 dll 022 “ b2ldi2Ci2d2l 
= ^11022^22 — diia22d2ldi2 — 62 lOl 2 d 220 ?ll — d\iC22b22 
+ 622 ^ 11^21012 + 621 ^ 12^11022 

= ^^1(022^22 — C22622) + ^11012(622^21 — 621(^22) + ^12^11(621022 — 022^21)- 

Matrix L is symplectic, therefore the following identities hold (see, for in¬ 
stance ini): 


aFc = c^a, b^d = d'^b, d^a — b^c = E = 


1 0 
0 1 


The hrst identity is equivalent to equality: 

ai20ll + 022021 = O12O11 -I- 022021- 

Similarly, the second matrix identity is reduced to equality: 

612^11 + 6220^21 = ^12611 + 0^22621. 

The third matrix identity gives the following relations: 

diioii -I- d2ia2i — 611C11 — 621C21 = 1 

<^llOl2 + <^21022 ~ 611C12 — 621C22 = 0 

<^12011 -I- 0^22021 — 612C11 — 622O21 = 0 

di 20 i 2 + 0^22022 ~ 612 C 12 — 622 O 22 = 1 


(4) 


(5) 
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Now, taking into account relations Q, the second and the fourth equalities 
in ([^, the expression for A can be transformed as follows: 

^ = d\i{^+bi2Ci2—di2ai2)+diiCi2{hiidi2—hi2dii)+di2dii{di\ai2—biiCi2) 
= dll + dlibi2Ci2 — diidi2ai2 + (iiici26iidi2 — diiCi2bi2 + ^11^12012 

— di2dii6iici2 = dll / 0- 

Thus, linear system (|^ has a unique solution {u,v) at the given {u,v). So, 
u = 0, V = 0 only if {u, v) = (0,0) and P intersects D at the unique point. 

■ 

The Assumption!^ says that / is degenerate since generically two smooth 
curves in a 4-dimensional manifold do not intersect. This assumption se¬ 
lects a codimension 2 set of diffeomorphisms in the space of all C'^'-smooth 
symplectic diffeomorphisms on M. Indeed, when a diffeomorphism with a 
homoclinic orbit to an 1-elliptic fixed point is perturbed within the class of 
smooth symplectic ones, for a perturbed /' the fixed point p' persists and its 
type is preserved. Therefore, due to transversality condition, the intersection 
of perturbed and W^{p') persists as well, but the intersection point 

does not give generically a homoclinic orbit to p' : with backward iterations 
of f the orbit through the intersection point can be either a heteroclinic 
orbit connecting p' and some invariant curve on W‘^{p') or some other or¬ 
bit wandering near (recall that there are instability regions on W‘^{p'), 
the orbit returns to W^{p') staying within 3-dimensional Vb™(p'), thus it 
is locked between unstable 2-dimensional manifolds of invariant curves on 
W^{p')^ since they locally divide W^'^{p')). 

Nevertheless, if we turn to the related 3 d.o.f. Hamiltonian system with 
a periodic orbit C of 1-elliptic type (or it can be called to be of the saddle- 
center type), then such an orbit belongs to a smooth symplectic cylinder of 
periodic orbits of the same type. So, if C has a homoclinic orbit, then for 
the related close levels of Hamiltonian on the cross-section to C one gets a 
one-parameter family of symplectic Poincare maps. Thus, if Hamiltonian 
itself depends on a parameter e in a generic way, then first return map for 
C, derived by a homoclinic orbits to it, unfolds to a two-parameter family of 
symplectic maps and hence any close smooth 1-parameter family of smooth 
Hamiltonians also has a 1-elliptic periodic orbit with a homoclinic orbits to 
it. Thus, this phenomenon is generic for generic 1-parameter unifoldings of 
a Hamiltonian with such the structure. 

Now we return to the problem under study. In a neighborhood of ho¬ 
moclinic point q let us consider 2-dimensional symplectic disk S through q 
being the transverse intersection of extended 3-dimensional center-unstable 
manifold IT™(p) with 3-dimensional center-stable manifold IT“(p). Below 
we shall prove the existence of smooth stable and unstable manifolds for any 
KAM-curve on lying in a sufficiently small neighborhood of p. All stable 
manifolds belong to IT“ and all unstable manifolds belong to Hence, 
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they intersect with S. The hrst statement concerning this intersection is the 
following: 

Lemma 2 Disk E contains two Cantor sets of smooth closed curves Wui'j) 
and Wsij) being, respectively, traces of the related stable W^{'y) and unstable 
manifolds of invariant KAM-curves 7 G W^. For a fixed invariant 
curve 7 integrals of 2-form oj over disks -D“(7) and D^{'y) bounded by Wui'j) 
and Ws{'y), are equal: 



Proof. The existence of stable manifold W^{'y) and unstable manifold 
1 T“( 7 ) of invariant KAM-curves 7 G W‘^ will be proved in Appendix. 

The transversality condition implies the intersection of VK™ with 1T“ 
near q to occur along a smooth 2-dimensional disk E. For every invariant 
curve 7 in its stable manifold being extended by in a hnite number 
of iterations reaches a neighborhood of q and transversely intersects E within 
VF“ ( 7 ) along closed curve Ws ( 7 ), the trace of IT® (p) is point q itself. Traces 
on E of W'^{p) and VF“( 7 ) in VF'^“(p) are respectively point q and curve 
Wuij). 

Consider now a piece-wise smooth 2-dimensional surface made up of a 
piece of the lateral side of the cylinder between and E, the piece 

of bounded by 7 and disk E. Integration of the form over this surface 
is reduced to the difference of integrals over the disk in and that over 
disk in E bounded by Ws{'y), since the integral over lateral side is equal zero 
(it is a Lagrangian submanifold). This gives the equality of integrals in the 
statement of the Lemma. Similarly, we get equality of integral over disk in 
W^, bounded by 7 , and integral over disk in E, bounded by Wui'j)- H 

3 Linearization and scattering map 

The genericity Assumption is formulated using scattering map S. In this 
section we will construct this map which acts on tangent plane TpW^. Scat¬ 
tering map is an analog of the scattering matrix for a Schrodinger type 
equation |42j . For the problems of the homoclinic dynamics related with 
non-hyperbolic equilibria this map was hrst introduced in [23]. Far-reaching 
extension of this map for a normally hyperbolic manifold in a Hamiltonian 
system was obtained in m- 

Consider hrst the linearization of the family of diffeomorphisms /” at 
homoclinic orbit F = n G N | qn+i = f{Qn), Qo = q}- This linearization 
is a sequence of linear symplectic maps = Df\rp ^ : Tq^M —)■ Tq^j^^M 
and hence limL^ = Dfp as |n| —>■ 00 . Since f^{q) —>■ p as n —)> ± 00 , there 
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exists an integer N large enough such that given a neighborhood V oi p one 
gets /”(g) G V for all |n| > N. 

In neighborhood V we choose a symplectic chart where hxed point p is 
the origin, then map / is in the standard form "linear diffeomorphism plus 
higher order terms". After a linear symplectic change of variables the linear 
part of the map can be transformed to the block-diagonal form: 


Xl 

yi 

Ul 

Vl 


px + ... 

p~^y + . 


COS a 

sin a 


— sin a 
cos a 


( 6 ) 


+ ..., 


with 0 < /i < 1, dots mean terms of the order 2 and higher. In these coordi¬ 
nates the linearization of this discrete dynamical system at the homoclinic 
orbit r is given as follows: 



— PnCn 

— p Pn QnCn 


(7) 


where (n = {^n, Pn, XnV = {^n,Pn, xli, XnV is Coordinate 4-column vector in 
the tangent space at the point Qn = {xn,yn,Un,Vn)', Ra denotes the rotation 
matrix through angle a: 


R 


a 


COS a — sin a 
sin a cos a 


Pn, Qn are 1-row matrices, Wn is (4 x 2)-matrix. Since Pn,Qn,Wn are of 
at least order 1 at (0,0, 0,0) and (x„, ?;„) decay exponentially fast to 

(0,0,0,0) as |n| —)■ oo, for these matrices the following estimates hold for 
\n\ > N and some positive C depending on N and on size of the neighbor¬ 
hood V: 

\\Pn\\ < C/r'”', IIQnll < 

where 0 < p < pi < 1. Take |n| > N and denote 


Sn = 


E 2 0 
0 Rna 


where E 2 ''\s 2 x2 identity matrix. 

Consider now the case n > N and perform in the system ([^ a sequence 
of nonautonomous (with "time" n) symplectic changes of variables (n = 
Sn-N'ipn, where V'n = {in, Pn, Xn), and consider ([^ in the rotating coordinate 
frame. This change of variables allows one to exclude asymptotically the 
rotation in coordinates Xn = {xh,Xn) and prove that in new coordinates 
each invariant bounded sequence for the linear system obtained from ([^ has 
a limit as n —)• 00. 
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After the change system Q casts as follows (we hold previous notations 
for variables): 

{ Cn+l ~ h'Cn “1“ F'nCn 

Vn+l — Vn GnCn ( 8 ) 

Xn+1 — Xn “ 1 “ HnCm 

where Fn, Gn are again 1-row matrices and Hn is 4 x 2-matrix. For these 
matrices estimates similar to those for matrices P^, Qn and Wn are valid. 
Sequence {(^njhroXn)} is called the solution of the system or the invariant 
sequence, if equalities (|^ are satished for all n G Z+. The following lemma 
is valid. 

Lemma 3 There is an integer N > 0 large enough such that for any given 
G M, x+ G a unique solution {(Cm hro Xn)}; n > N , exists for the 
system (® such that for this solution the boundary conditions are satisfied: 
Cat = Xn -t X+; lCn| 0, |r/„| 0 as n -hoo. 

Proof. Similar to |24| . instead of system ([^ consider a system of difference 
equations 

' n-l 

Cn = + E l^"-^-^FsCs 

s=N 
oo 

Vn = ( 9 ) 

s=n 

oo 

Xn = X+ ~ E FlsCs- 

, s=n 

Note that any solution of this system obeys the boundary conditions in the 
statement of the lemma. Let us show hrst that the solution of the system 
is also the solution of the system ([^ and vice versa. Indeed, the following 
equalities hold: 


Cn+l = + E 

s=N 

( n—1 

+ E ) + FnCn = TU + FnCn 

s=N / 

oo oo 

Vn+l = - E l^"~"GsCs = GnCn-GnCn- E T"~"GsCs 


s=n+l 


5 = 72+1 


GnCn + h ^ ^Ggtfs I ~ + GnCn 


s=n 

oo 


Xn-l-1 — X+ 'y ^ FgQs — y^_|_ y ^ HgQg -|- FlnCn — Xn T HnQn- 

s=n-|-l s=n 
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So, if the sequence {{Cm Vn, Xn)} solves ([^, then it satishes ([^. The converse 
assertion is given as n —)• oo by the consecutive application of ([^ to an initial 
point. 

Thus, one needs to prove the existence of solutions for system ([^. To 
do this, we use the contraction mapping principle. Denote B the Banach 
space of sequences C = {{CmVnjXn)} uniformly bounded on [A^,+oo) with 
the norm 

IICII = SUp(|Cn|,|??n|,||Xn||). 
n>N 

Right hand sides of ^ dehne operator T : r[(^] = ^ on R. At the hrst step 
let us verify that T is dehned correctly, that is T[B] C B, here x+> 

N are considered as parameters. Recall that for Fn, Gn, Hn the following 
estimates are valid: ||Tli|| , HGnll j ||-f^n|| < Q < v < 1. Here C depends 
on N, but is hnite for a hxed N. Denote k = max{/i, < 1. Then one 
proceeds as follows: 


71—1 


le'nl < + IICII + IICII CK^-\n - N), 

s=N 

°° °° 2n 

li„l < IICII E = c IICII k'-" E «"■ = c IICII 


— 


= C\ 


^n+l 


Ili„ll<lli+Il + C||C|IEK‘ = I|X+I| + C||ci|j 


— K 


Thus, the sequence C is uniformly bounded on [A^,+00), so the operator T 
is dehned correctly. 

Next we prove T to be a contraction map: 

n—1 


|C^ - C^l < IlCi - C2II C- ^ = IICi - C2II CK^-\n - AT), 


s=N 

00 


K-vl\ < 


IICi-C 2 ||c^, 


.s+1—n_s _ 


^ 2 n 


k* = ||Ci-C 2 ||Ck 


1—n 


1 - 


r<^n+l 

11 !_ 11 ^ 11 !_ 11 r\j 

= IICl - C2II ^ _ ^2 - ll*^! “ ^2|| ^ _ ^2’ 

\xi-xl\\ < IICi-C 2 ||cE'^‘ = IICi-C2l|:j— < IIC1-C2II2—. 

i — Ki X — Kj 


These estimates show that T is contracting for N large enough and n > N. 
Thus, for any hxed G M and G there is a unique solution C(C*^7 X+) = 
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{Cn(?°,x+)} = {(?n(^°,X+),??n(?°,X+),Xn(?°,X+))} for the system (g such 
that ^n{^^,X+) = The estimates above also show that |^„|, |r/„| and 
WXn — X+ll tend to zero as n —)• oo. ■ 

For the further purposes one needs to prove some linearity relations for 
solutions of system g. 

Lemma 4 Solutions C(^^>X+) of the system g satisfy the following linear¬ 
ity relations: 

I. aeo, x+) + c(?o > 0 ) = c(eo+x+); 
c(eo, 0) + c(eo, 0) = c(eo + o), c«, o) = «c(eo, o); 

///. C( 0 ,ax+ + / 3 xT) = «C( 0 ,x'+) + ^C( 0 ,xT)- 

Proof. To prove the first equality consider the function 


— {Ai(n)} — C(? 0 ) X+) + C(Co) 0) — C(^o + X+) 

(/ u^o,x+)+u^o,o)-ueo+eo,x+) \) 

= S Vn(^o,X-h) +hn(^o, 0 ) -r}n(fo + ^o,X-h) > 

[ \ XnUo, X+) + Xni^o, 0) - Xn(?o + , X+) / J 


This function is a solution of the system g with boundary conditions (0, 0). 
Indeed, consider the following systems with boundary conditions (^g,x+)) 
(?o>0) and (^g + ^^',x+), respectively: 


/ 

?n(?o,X+) 
^ ??n(^ 0 >X+) 

Xn(^ 0 >X+) 


71—1 

s=N 

+00 

- j: h^^"-"GsCs{eo,x+) ( 10 ) 

s=n 

+ 00 

X+ - E HsCsif,o,X+), 

s=n 


Cn (Co 5 0) 
^ ^r!-(Co) 0 ) 

Xn(Co ) 0 ) 


71—1 

E /^”-'-^T.Cs(Co,o) 

s=N 

+00 

- E /i^+'-"G.O(Co,0) 

s=n 

+ 00 

X+ - E ^.6(Co,o), 

s=n 


( 11 ) 


Cn(Co + Co,X+) 
^ ??n(Co +Co,X+) 
Xn(Co + Co) X+) 


^n-N^O + ^ 

s=N 

+ 0O 

-Ei^*+'“"a6(C(, + Cx+) (12) 

s=n 

+O0 

X+- E i^.6(Co + Co,x+)- 

s=n 
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Summing related equalities from ( |10[ ), © and subtracting ( |12[ ) we get: 
^n(?0) X+) + ^n(^0,0) — Cn(?0 + ^0 J X+) = + ^0 “ (?0 + ^o)) 

n—1 

+ ^ X+) + Cs(?0! 0) “ Cs(^0 + ^0) X+)) 


s=N 


n—1 




s=N 


Vn(^0, X+) + Vni^O, 0) - r?n(^o + , X+) 


+00 


= -^/^-^^-^-^^G,(Ueo,X+} + Ueo,0)-Cs(eo + ^o,X+)) 

s=n 

+00 


Xni^O, X+) + Xn(?0 ) 0) “ Xn(^0 + ?0 ) X+) 

+ 00 

= (x+ + 0 — x+) — ^ HsiCsi^o, X+) + Cs(^m 0) — Cs(Co + ^ 0 ) X+)) 

s=n 

H-oo 

= -Y,HsA,{s). 

s=n 

The equalities obtained imply that Ai satisfies ^ with boundary conditions 
(^*^,X+) = (0,0). Since the solution of ^ with given boundary conditions 
is unique, then Ai = 0 and therefore the relation I is valid. 

Relations II and III are proved in a similar way, if instead of Ai one 
considers A 2 and A 3 , respectively: 

A 2 = C« + 13 ^ 0 , 0) - aC(^o, 0) - /3C(eo, 0), 

As = C(0, ax'+ + /3x+) - aC(0, x'+) - ^C(0, X+)- 

Lemma has been proved. ■ 

Similar lemmas hold for n < —N. 

3.1 Geometry of linearized map 

Now let us present a geometrical interpretation of the results obtained. To do 
this, we introduce a countable set of linear symplectic spaces \n\ > N, 
with coordinates (Cn, ??n, Xn? Xn); linear symplectic maps Cn ■ Xn —>■ 
A„+i, n > A, defined by (^. If we fix x+ and vary ^ 0 , Co = Co + Cm Co ^ M, 
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then due to Lemma the related solutions of ([^ define an affine straight 
line in Xtv (in fact, they are initial points of these solutions) and hence in 
any X^, n > N. These straight lines have the characteristic property that 
any solution which passes through this line in decays exponentially as 
n —>■ oo: Cn —>■ 0, —)• 0, \\xn ~ X+ll 0- addition, if we fix not x+ but 

only the value 2/ = = llx+lPi then in every n > N we get a 

2 -dimensional cylinder C^{I) formed of those straight lines in through 
which solutions asymptotically satisfy to ||Xn — X+ll 0) each straight line 
on (/) corresponds to the unique value on the circle 21 = Xi + xi 
asymptotic phase). Varying /,^o defines a linear 3-dimensional subspace 
of bounded solutions in Vjv, and hence in X^, which in turn foliates on 
the cylinders C^{I). Such a cylinder shrinks to the straight line (7^(0), as 
/ —>■ 0, this straight line just corresponds to solutions with x+ = 0. 

Now let us turn to the initial linearization problem along homoclinic or¬ 
bit r for diffeomorphism /. To derive results described above, we performed 
the sequence of linear changes of variables that allowed us to prove for any 
bounded solution the existence of an asymptotic phase. In the initial coordi¬ 
nates all objects found preserve: for any point in the related tangent space 
Tq^M we have 3-dimensional subspace of bounded as n —)• oo solutions C'^ 
(we preserve the same notations for similar objects) which are foliated into 
cylinders (/) (it is worth mentioning that the value of I does not change 
when returning to the initial coordinates), foliations into straight lines, etc. 
It is evident that in fact C'^ is nothing else as tangent space 

The same picture takes place for X^ with n < —N^ the only difference 
is that one needs take limits as n —?• —oo. Here we also have cylinders C~, 
straight lines, 3-subspaces C~ = Tq^W^'^, and so forth. 

For tangent space Tq(^_j^-^M and Tq(^j^'^M we have linear symplectic map 
S = Df'^^ : Tq^_j^'^M —)• Tq(^f^'^M calculated at the point q{—N). This map 
transforms C~ to a 3-dimensional subspace in Tq(^^'jM which transversely 
intersects the straight line C')^(O) = being the tangent space to W^{p). 

3.2 Scattering map 

Now we are ready to construct the scattering map S : TpW^ —)• TpW^. Take 
any point s G TpW^. Fixing this point defines the unique straight line in 
Tq(^_]\i'jM of the foliation defined in C-i\f = whose points are 

asymptotic to s as n —>■ —oo. Let us apply linear map Df'^^ to points of 
this line. We get the straight line in Tq(^j^'^M which is transversal to 3-plane 
Cm = due to transversality condition. Thus, the line obtained 

intersects this 3-plane at the unique point through which a unique line of 
the foliation defined in plane Cm passes. Denote si G TpW’^ that unique 
point which is the limit as n —>■ oo for all sequences starting on this line. We 
set S{s) = Si (Fig. [^. 

Let us verify that S' is a linear map. It is clear that S'(O) = 0. Indeed, 
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Figure 2: Scattering map construction 


for s = 0 the corresponding straight line in 3-plane C-n is the tangent line 
to VF“ in tangent space Its image under S = is a straight 

line in which is transversal to due to Assumption and 

intersects it at the origin of Through the origin the unique line from 

the constructed foliation passes: the tangent line to which corresponds 
to Si = 0 in TpW^. 

Denote Ig that straight line in T-at which consists of points through 
which solutions pass tending to s as n —?• —oo. Using the linearity relations 
I and II we get the following representation for the solutions of system (|^: 

Cn(?°, Ax-) = Cn(0, Ax-) + Cn(^°,0) = ACn(0,X-) + Cn(?°,0) for any A G M. 

To hnd the image S{Xs) we act by on Xlg + Iq (here Iq is the tangent 

line to VF“ at point q{—N)). In we get two vectors, through each such 
vector a unique straight line passes: these lines are XI s and Iq, respectively 
{Iq is tangent line to at point q]\f). Thus, S maps As to Asi -|- 0 = Asi. 
Similarly, for the sum s' -|- s" we get relations for corresponding X-; X- • 

Cn(?°) X- + X-) = + 0, X- + X-) = 

= Cn{C^, 0) -|- Cn(0, X- + X-) = Cn(C°, 0) -|- Cn(0, X-) + Cn(0, X-)- 

In this case we act by on Ig/ -|- Ign + Iq- 

The next proposition characterizes map S. 
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Proposition 1 Map S : TpW^ —)• TpW^ is symplectic. 

Proof. Choose any two vectors in the symplectic plane TpW^. These vectors 
define two straight lines from the foliation in T-at. Take then two vectors 
vi,V 2 in C-N = Tq(_Ar)lT'^“ corresponding to these lines: origins of vectors 
coincide with zero point of Tq(_Ar)lT™ and ends of the vectors belong to 
corresponding line. Skew-scalar prodnct being the restriction of 2-form 
on tangent space C-isf, does not depend on vectors we choose. Indeed, 
difference of vectors corresponding to the same line is vector lying in Vt which 
is zero vector for skew-scalar prodnct (snch vectors shrink exponentially in 
backward iterations). 5-images of these two straight lines are two straight 
lines in which are transversal to snbspace Cm- The intersection 

of the lines with Cm gives two vectors T{vi),T{v 2 )-, whose origins coincides 
with zero oi Cm = Tg(7v)kP“) for specified vi,V 2 - Since 5 is linear symplectic 
map, then the skew-scalar prodnct is preserved. Now we have two straight 
lines from foliation in mathcalLM and again skew-scalar prodnct of vectors 
corresponding to different lines does not depend on exact vectors we choose. 
Bnt this prodnct is eqnal to skew-scalar prodnct of vectors vi,V 2 and does 
not change in forward iterations. Therefore skew-scalar prodnct in limit in 
TpW^ eqnals to skew-scalar prodnct of initial vectors in TpW^. ■ 

Linear symplectic map S we call the scattering map. 


4 Homoclinic orbits to invariant KAM-curves 

To prove Theorem we assnme for diffeomorphism / Assnmptions and 
to hold. Thns, according to Section [^manifolds lT'^“(p) and lT“(p) intersect 
at homoclinic point q transversally, and therefore along a symplectic 2-disk S 
(we may regard q = and disk S to belong to a neighborhood of fixed point 
p). The idea of the existence proof for homoclinic orbits to invariant KAM- 
cnrve 7 G IT'^ is the following. Let V he. a. snfficiently small neighborhood of 
point p. For each KAM-cnrve 7 C H V its action is defined according to 
the Stokes theorem as the integral of 2-form Q over that disk in VF'^ whose 
boundary is curve 7. Curve 7 has its local stable manifold which can be 
extended by a finite number of iterations of map till the manifold reaches 
the neighborhood of homoclinic point q staying inside of LF“(p). Therefore, 
this manifold intersects transversally within W^{p) disk S along a closed 
curve Ws{'^)- Similarly, unstable manifold of the same curve 7 under the 
action of / reaches the neighborhood of q staying inside W^^{p) and hence 
intersects S along a closed curve Wu{'y). Two obtained curves on S have 
the same value of action as follows from lemma Thus, two disks in E 
bounded by Wsi'j), Wui'j) are of the same area and have common point q 
lying inside both of them. Hence, the intersection of curves Wg^'j) and vJui'j) 
is not empty and consists of at least two different points, homoclinic orbits 
to 7 pass through the intersection points (see Fig. |^. 
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( 1 ) 


( 2 ) 


( 3 ) 


Figure 3: Possible intersections of VF^(7), on S 


The problem here is that we do not know a precise information on this in¬ 
tersection: how many points does it contain, if the intersection is transverse 
or not, etc. All these questions are relevant for the further study of nearby 
dynamics. In the case of an integrable diffeomorphism (when an invariant 
w.r.t. the diffeomorphism / smooth function exists) these two curves coin¬ 
cide, since both of them belong to the same level of the invariant function 
whose restriction on S near q usually forms a connected closed curve. 

Provided that our Assumptions hold, we shall prove for any KAM curve 
on in a sufficiently small V with a given value of action I, the intersection 
to consist of exactly four points and it is transverse at each of these points. 
To prove this we shall connect intersection properties for curves Wui'j) and 
Wsi'j) on S with intersection properties of related ellipses E and S{E) with 
the same action I in tangent plane TpW^. The genericity Assumption implies 
that these ellipses have the same center and the same area and intersect 
transversally at exactly four points. This property will be carried to the 
intersection of Wui'j) 

To prove the intersection of theses curve as in Fig. (left panel), we 
transform / in neighborhood V of fixed point p to the normal form ( |15[ ) (see 
Appendix) up to the third order terms and consider hrst the truncated map. 
For this map two local functions xy and are local integrals up to sixth 

order terms, but what is more important for our goals, circles x = y = 0, 
-t- = c on are invariant curves for any positive c small enough and 

stable and unstable manifolds of these curves have the representation y = 0, 
= c and x = 0, = c, respectively. Since the whole map 

differs from the truncated map by terms of the fourth order and higher, then 
for a given positive constant c small enough invariant manifolds of KAM- 
curves for the truncated and full maps are at least C^-close in F, due to 
Fenichel theorems. According to these theorems |16| . as diffeomorphism is 
C'^'-smooth, then invariant cylinders of KAM-curves are (7^“^-smooth (see 
Appendix). Also we shall suppose, without loss of generality, that both 
points q- = qi—N) and g_|_ = q{N) belong to V. 
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For diffeomorphism (15) for n> N, due to properties of functions (^, '0, 
system ([8|) has the linearization matrices for / along the homoclinic orbit 


(+ • 
0 

0 

V 0 




Pn 

-\l + 
In 
Sn 


Qn 

0 

cos a + 
sin a + • 


Tn 

0 

— sin a + 
cos a + • 




/ 


where dots and Pn^Qn^fn^ln, Sn mean terms tending to zero exponentially 
fast as n —>■ +oo. The form of this matrix implies that 3-dimensional plane 
r/ = 0 in the tangent space at homoclinic point q{N) under the action of this 
linear map is transformed to 3-dimensional plane = 0 in the tangent space 
at homoclinic points q{N -|- 1) = f{q{N)), etc. As Lemma implies, for a 
fixed /, (x^)^ + (x^)^ = 27, we get in a cylinder in 3-dimensional 

plane rj = 0, consisting of solutions for the system ([^ which asymptotically 
tend to the circle (x^)^ + (x^)^ = 2/ in TpW^. Intersection of this cylinder 
with tangent plane to at point q{N) is an ellipse. Similar cylinders and 
ellipses are obtained, if one considers linearization of diffeomorpsim along 
homoclinic orbit for n < —N as re —)> —oo. 

Global symplectic map Q = transforms a neighborhood of homoclinic 
point q{—N) to a neighborhood of homoclinic point q{N). In normalized 


coordinates (15), homoclinic points have coordinates: q{—N) = (0,?/i,0,0), 
q{N) = (xo, 0,0,0). Therefore, symplectic map G has the following local 
representation 

F{x,y- yi,u,v), 

G{x,y- yi,u,v), 

P{x,y- yi,u,v), 

Q{x,y- yi,u,v), 


X — Xo = 

y = 

u = 


V 


dx A dy + du A dv = dx A dy + du A dv, 

and all functions F,G,P,Q vanish at the point (0,yi,0,0). Transversality 
condition at point q{N) means the tangent vector to W^, that is (1,0, 0,0), 
be transverse to tangent plane to IT™ being ^-image of 3-disk x = 0. This 
implies that determinant 


det 


d{G,P,Q) \ 
diy - yi,u,v)J 


calculated at point (0,yi,0,0) does not vanish. In virtue of Lemma the 
same transversality condition holds at q{—N): tangent vector to 1T“ (i.e. 
(0,1,0, 0)) and tangent plane to VF™ (^-pre-image of y = 0) are transverse, 
this is equivalent to inequality Gy 7^ 0 at (0, yi, 0,0). 2-disk S = at these 
coordinates is the intersection of ^-image of local 3-disk x = 0 near point 
q{—N) and local 3-disk y = 0 near q{N). Then one has a representation 
for S_ = ^~^(S_|_) in the form x = 0, y — yi = T(re,u), and for S_|_ in the 
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form X = ^{u,v), y = 0. In particular, since map Q is symplectic and the 
restrictions of symplectic 2-form II to symplectic disks S_ and S_|_ are duAdv 
and du A dv, respectively, then the restriction of map Q, Qy. '■ —)• 

is symplectic and has the form 


u = aiiu + ai 2 V + ... ,v = a 2 iu + 022^ + ■ • •, (13) 


where dots mean terms of the second order and higher, matrix 


A = 


( ail 

Va2i 



is symplectic, here this means to be uni-modular: det A = 1. In fact, matrix 
A depends on integer parameter N, A = A]\f, since our choice of the homo- 
clinic points q{—N),q(N) and hence S_, and related tangent planes to 
them depends on N. But for any N large enough the representations for 
these tangent plane are similar and coordinates of them are coordinates on 
TpW^. As iV —>■ 00 the related tangent planes to S+(A^) tend to TpW^. This 
implies 


Lemma 5 For N large enough matrix Ain is not a rotation matrix. 


Proof. To prove Lemma we shall show that its conclusion follows from the 
transversality Assumption and genericity Assumption As follows from 
Lemmas 3, 4, Am tends to matrix Aoo being the coordinate representation 
of scattering map S. ■ 

Consider symplectic disk S_ through the point q{—N). In normalized 
coordinates it has a representation x = t), y = '^{u,v). Hence the tangent 
plane to it at q{—N) has a representation ^ = 0, 77 = + b^. This implies 

this plane to intersect transversely any Lagrangian cylinder C~{I) and these 
intersections form a foliation of the plane into ellipses. Similar foliation 
into ellipses exists in the tangent plane to through the point q{N), it is 
generated by intersection of this plane with cylinders C^{I). Differential of 
the global map Q = is a linear symplectic map and its restriction to 
S_ is a linear symplectic map Lm : Tq(_jv)D_ —)• Tg(^m)'^+- If "w® Hx I, then 
related ellipses E-{I), E+{I) in S_,S+, respectively, have the same area 
and Lm{E-{I)) intersects E^(I) at four different points transversely, due to 
Assumption 

Let us hx a sufficiently small a neighborhood of hxed point p on the 
center manifold W^. Its smallness is controlled by a parameter e. To this 
end we, following [36], we introduce symplectic polar coordinates on = 
{x = 7/ = 0} in the neighborhood of hxed point 

u = eV^cos9, v = eV^sm6. 


Then we get a symplectic map on W^: 

I = I + Oie^), 

e = 0 + a + i^e‘^I + Oi£^). 
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(14) 


According to the Moser theorem |3U], there is an Jq > 0 such that for s 
small enough and any given I, 0 < I < Iq, such that the number a + 
21/8^1 is Diophantine, there exists an invariant curve which is e^-close in 
topology in the space of curves I = r{9) to curve + v'^ = 2e^I. The map 


(14 1 is the restriction of the initial map on a neighborhood of p in and 


normalized up to the third order terms. Thus each invariant curve on the 
center manifold has two invariant Lagrangian cylinders being its stable and 
unstable manifolds, they are C^-close to cylinders v? + = 2 e^/, ?/ = 0 , or 

X = 0 , respectively, of the truncated map. 

Let us verify that traces of 1 T^( 7 ), corresponding to invariant 

KAM curve 7 on TpW^ with action also intersect on S_|_ transversally 
along four points. To this purpose, we consider the restriction of Q on S_ 
near point q{—N) with values on disk S_|_ near point q{N). Fix in V some 
neighborhood of p on defined by e small enough, and let 7 be some 
KAM-curve in this neighborhood. This defines some I. The restriction of Q 
on S_ is a two dimensional symplectic map (13). Since the coordinates on 
S_ and are {u,v), after the change of variables (14) where e and I are 


considered as parameters, we come to the system for intersection points of 
VF®( 7 ), Taking into account that these manifolds of the same KAM 

curve, we get the value I be the same and then we have: 

ey/^cosO = eoiiV^cos^ + eai2V^sin0 + C)(e^), 
sin 0 = 6021 \/^ cos 6 + 6022 \/^ sin0 + 0(6^). 

Dividing the equations on squaring both sides of each of the equalities 

and sum them we get the equation for 9 corresponding intersection points: 


1 = (oii COS0 + 012 sin0)^ + (021 cos 9 + 022 sin0)^ + 0{e). 


This equation have precisely four simple roots, if matrix A is not a rotation 
matrix m- Therefore, due to implicit function theorem, traces of cylinders 
of truncated map on disk S intersect transversally along four points. This 
implies that C^-close traces of cylinders of full map, that is curves Wui'j) 
and Wsij), also intersect transversally along four points. Theorem has 
been proved. ■ 


5 Homoclinic orbits to saddle periodic orbits on 

As is well known, a sufficiently smooth 2-dimensional symplectic diffeomor- 
phism near its generic elliptic fixed point O (some Birkhoff coefficient in the 
normal form does not vanish) is a twist map with respect to action-angle 
variables given by symplectic coordinates of the Birkhoff normal form. In 
particular, this implies the existence of a positive measure Cantor set of in¬ 
variant KAM curves with Diophantine rotation numbers accumulating at p 
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|3b) . For our case as such diffeomorphism we have the restriction of / onto 
Since is a smooth normally hyperbolic invariant submanifold for 
/ the Fenichel results usms] apply that gives a smooth foliation of and 
into smooth curves (see Appendix). This smooth foliations allow us to 
define two smooth maps t S, Fg : —)• S. 

Let us fix some such invariant KAM curve 7 with a Diophantine rotation 
number. Near such the curve there is a positive measure set of other smooth 
invariant curves accumulating at 7 in at least C^-topology. Another conse¬ 
quence of the KAM theorem, twist condition is that for any rational num¬ 
ber p/q with incommensurate integers p,q there are at least two g-periodic 
points. Generically, these two periodic orbits are elliptic one and hyperbolic 
another. Moreover, if this diffeomorphism satisfies some additional genericity 
condition (sometimes, it is called as the Birkhoff genericity [?]), then stable 
and unstable separatrices of the hyperbolic orbit intersect transversely along 
related homoclinic orbits in This allows, in particular, to construct a 
“fence” made up of stable and unstable separatrices separated one invariant 
curve from another one. 

This can be done in the following way. Let us assume g = 2 to be 
definite. Take one hyperbolic 2-periodic point mo and let mi be its first it¬ 
eration: mi = /(mo). Suppose that unstable manifold VF“(mo) transversely 
intersects stable manifold of VF^(mi) and stable manifold of IT^(mo) trans¬ 
versely intersects unstable manifold of lT“(mi). In this case, due to so-called 
lambda lemma the topological limit of LF“(mo) contains LF“(mi) and 
vise versa. Thus we get some closed invariant set 5^^ made up of these curves 
and their closures. Similar set is formed by stable manifolds (see Fig.??). 

Take and for every its point consider the related unstable leaf of the 
unstable foliation in Then map F^ transforms set to the homeomor- 
phic set in S. Similar set in S is obtained from 5^^ using Fg. Now choose any 
invariant KAM curve in in a neighborhood U where Theorem [^applies. 
Choose a sufficiently close invariant KAM curve 71 such that on S related 
traces {'y), {ji) intersect transversely traces tc“(7), tc“(7i). Then we 
have on S two annuli: Ag bounded by {'y), {'ji) and Au bounded by 
w'^{'y), {'yi). These annuli intersect each other in such a way that each 
boundary curve of one annulus intersects every boundary curve of another 
annulus transversely. Since the restriction fc of / on W‘^ is a twist map, then 
invariant KAM curves 7,71 have different rotation numbers p, pi. Thus there 
are periodic orbits inside the annulus between 7,71 corresponding to some 
rational p < p^ < pi, = p/q- If /c is Birkhoff generic, then the half of 
these periodic orbits are hyperbolic Birkhoff g-periodic and its stable mani¬ 
folds form a fence 5^^ in C S. This fence separates Ag in the sense that if 
we take two points on different boundary curves of A^, then any path going 
from one point to another one will cut '^g. The same holds for in A„. 
This implies 


23 



Figure 4: A fence on S made up of stable and unstable manifolds of a saddle 
periodic orbit. 
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Theorem 2 The sets du interseet, henee there are Poineare homoelinie 
orbits to a saddle hyperbolie periodic orbit on . 

It is clear that in fact there are countably many such Poincare homoclinic 
orbits. It is impossible to assert that they are transverse or tangent since 
this cannot be caught by such considerations. 

6 Multidimensional extension 

The problem we have studied possesses a multidimensional extension. The 
tool to get this extension are essentially the same, so we present only the 
related set up and formulations. In a smooth symplectic manifold (M, O) of 
dimension 2n + 2 we consider a symplectic diffeomorphism / that possesses 
a fixed point p of the elliptic-hyperbolic (2, 2n)-type. The latter means this 
point has the linearization operator with the only pair of complex eigenvalues 
on the unit circle exp[±iQ;] and remaining 2n eigenvalues are off the unit 
circle and thus are met either in real pairs \pLk\ < 1, or in complex 

quartets pm exp[±iKm], exp[±mm],, Km / 0, vr. Here one has A:-|-2m = n. 
Such a fixed point has locally a smooth two dimensional center manifold 
corresponding to the pair exp[±iQ;] on which p is an elliptic fixed point 
and we assume henceforth it to be of generic elliptic type. Besides center 
manifold, through fixed point other smooth manifolds pass: n-dimensional 
strong stable and strong unstable 1T“ ones, as well as (n-|-2)-dimensional 
center stable 1T“ and center unstable 1T‘^“ ones. 

Analogs of three Assumptions 1-3 are 

Assumption 4 (Homoclinic intersection) Manifolds lT“(p) and IT^(p) 
have an intersection at some point q, generating thus a homoclinic orbit T 
to point p. 

Dimensions of stable IT® (n) and center unstable manifold 1T'^“ {n + 2) are 
complementary, this allows one to assume 

Assumptions (Transversality condition) The intersection of manifolds 
W^{p) and W^'^{p) at point q is transverse. 

Below we shall show that the linearized along the homoclinic orbit F the 
sequence of linearized map generates the linear symplectic scattering map 
S : TpW^ —7- TpW‘^. We assume this map being generic that means as above 
that the foliation into ellipses on the tangent plane TpW^ generated by the 
linearized map Dfp has the property: any ellipse E of this foliation satisfies 

Assumption 6 (Genericity condition) The intersection Er\S{E) is trans¬ 
verse and consists of four points. 

If these three conditions hold then the analog of the main theorem is valid. 
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Theorem 3 Let a -smooth, r > 6, symplectic diffeomorphism f on a 
C°° -smooth 2(n + 1)-dimensional symplectic manifold M with an elliptic- 
hyperbolic fixed point of the type ( 2 , 2n) obeys Assumptions 0 [ 11 @ Then 
there is a neighborhood U of homoclinic orbit T such that every closed in¬ 
variant KAM-curve on possesses four transverse homoclinic orbits in 
U. 

To prove this theorem we again first study the linearized nonautonomous 
problem given by the linearization of / on the homoclinic orbit T. Also, in 
order to avoid possible complications, one assumes in addition that orbit T 
leaves from p and enters to p along leading direction in 1 T“ and (one or 
two dimensional). 

Then, as above, we construct scattering map S acting on TpW'^ and as¬ 
suming Assumptions 4-6 to hold we prove the Theorem. This proof uses 
again that the transverse intersection of W'^'^ and near a homoclinic 
point g G r occurs along a 2 -dimensional disk S which belongs to both of 
them. Hence (n -|- l)-dimensional stable and unstable manifolds of any in¬ 
variant KAM curve 7 C when continuing by / in 1T“, respectively, 
intersect again S along closed curves re®( 7 ), 7 /;“( 7 ). Genericity Assumption]^ 
implies this intersection to happen transversely at four points through which 
homoclinic orbits to 7 pass. 

7 Appendices 

7.1 Straightening invariant manifolds 

In some neighborhood of the fixed point the symplectic diffeomorphism un¬ 
der consideration can be written as Q. In this form 1-dimensional stable 
manifold IT® is given as a smooth curve tangent to the x-axis (at point p) 
and 1 -dimensional unstable manifold 1 T“ is given as a smooth curve tan¬ 
gent to the y-axis. Center stable IT'^® and center unstable 1T“ manifolds 
are given as graphs of the functions y = F(x,u,v) and x = G{y,u,v) being 
tangent at p to 3-dimensional planes y = 0 and x = 0, respectively. Let us 
first straighten the curves IT®, 1T“: 

Lemma 6 Let in linear symplectic space (M^,H = dx A dy + du A dv) a 
smooth curve (x,y{x),u{x),v{x)) through the point (0, 0, 0, 0) is given, such 
that y(0) = u'{0) = x^(0) = 0. Then this curve can be transformed by a 
symplectic transformation {x,y,u,v) -A (^, 7 , cj) into the f^-axis. 

There are many of such transformations, for instance, this is one of them: 

' f = X 

V = y — y{x) — v'{x){u — u{x))-\-u'{x){v — v{x)) 

V = u — u{x) 

^ U = V — v{x). 
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All other transformations in a neighborhood of p we perform holding straight 
and W^. At the next step we straighten and 

Lemma 7 In some neighborhood of point p there exist symplectic coordinates 
(x, y, u, v), It = dx Ady + du A dv, such that submanifolds in these 

coordinates become flat, that is they are given as y = 0, x = 0, respectively. 

Proof. In principal, this lemma follows from the related result of the theory 
of symplectic manifolds (the relative Darboux theorem) |3]. But for the 
reader’s convenience, we present a direct proof. We follow the lines of the 
proof of the Darboux theorem given in |1]. 

In coordinates (|^ is expressed as y = F{x,u,v), where P(0,0,0) = 
Pj;(0,0,0) = Fn(0,0,0) = F^(0,0,0) = 0. Center unstable manifold in 
the same coordinates is given as x = G{y,u,v), G(0,0,0) = Gy(0,0,0) = 
Gm( 0,0,0) = Gt,(0,0,0) = 0. Take function pi = y — F as a Hamilton 
function and consider the related Hamiltonian flow. Since x = 1 then in a 
neighborhood of p small enough submanifold x = G{y, u, v) is transversal to 
flow orbits. We take this manifold as a cross-section to the flow. Denote 
qi{x,y,u,v) the time needed for the flow orbit through the initial point 
(xo = G{yo,uo,vo),yo,uo,vo) to reach the point {x,y,u,v). Then = 0 
for points on the cross-section and = 0 on IT“ since it is a level of the 
Hamiltonian. The Lie derivative of qi w. r. t. the vector field is equal to 
1. Therefore, Hamiltonian vector fields with the Hamilton functions pi,qi 
are commute and independent in a neighborhood of p. Thus, orbits of Re¬ 
action generated by these two commuting Hamilton functions give a smooth 
foliation into 2 -dimensional orbits near p and its leaves are transversal to 
2-dimensional submanifold being joint level of functions pi and qi. Next we 
take joint level pi = qi = 0 that is just locally W^. We introduce any local 
symplectic w. r. t. the restriction of 2 -form D on IT'’ coordinates {p 2 ,Q 2 ) 
near p. These coordinates are extended onto a neighborhood of W’ setting 
{p 2 ,q 2 ) constant along the whole 2 -dimensional orbit of the action through 
the point on IT'’ with coordinates {p 2 , ( 72 ) on it. ■ 

Remark 1 If and IT“ were previously made straighten, then one has 
F{x, 0, 0) = 0, G(y, 0, 0) = 0 and straightening IT'’®, IT'’“ preserves IT®, IT“ 
straighten. 

7.2 Normal form near 1-elliptic fixed point 

Here we shall derive the normal form up to the terms of third order for a 
smooth symplectic 4-dim diffeomorphism / in neighborhood V of its fixed 
1-elliptic point p. Without a loss of generality one may assume a G ( 0 , 7 r). 

Proposition 2 In some neighborhood V of fixed 1-elliptic point p there exist 
symplectic coordinates {x,y,u,v), H = dx A dy + du A dv, such that diffeo- 
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morphism f has the following form in these eoordinates: 


Xn+l 

Dn+l 

Un+l 

Xn+l 


+ aXnVn + b{ul + V^) + £> 3 ) 

h~^yn{l - axy - b{ul + vl) + O3) 

Un cos(a + u{u^ + Vn)) - Vn sin(a + u{u^ + v"^)) 
-HXnVniUn Sm a + Vn COS a) + (p{Xn, Vn, Un, Vn) 
Un sin(a + i'{u‘^ + v^^)) + Vn cos{a + v{u^ + f^)) 
+HXnyn{Un COSO - VnSma) + IpiXn, yn, Un, Vn), 


where u ^ 0 , functions tp, if are of the fourth order and higher at the origin, 
O 3 means terms of third order and higher at the origin. In these coordi¬ 
nates manifolds W^, coincide with x-axis, y-axis, respectively, that is 
the following identities hold: 0, 0, 0) = (^(0,y, 0,0) = 0,0,0) = 

V’(0,y,0,0) = 0. 


Proof. At the first step we straighten manifolds in the neigh¬ 

borhood of p (see Appendix). As the result, first two relations in (|^ are 
transformed to the form: 


Xn+l=Xn{yI -), yn+l=yn{y —)■ 


Next we apply the standard normal form method for symplectic maps (see 
for instance 0) We shall use such symplectic coordinate transformations 
which hold be straightened. Next we use complex coordinates 

instead of u,v in order to diagonalize the linear part of the third and fourth 
relations. Monomials of the second order and those of third order other than 
resonance monomials can be killed. 

Resonance relations for the set of eigenvalues (/r,e*“, e“*“) and in¬ 
teger vectors (mi, m 2 , ni, 712 ) have the form: 


y 

y 

y 

y 


mi—m2 —lgia(ni—n2) 
mi—m2-|-lgia(ni-n2) 
mi—m2gia(ni—n2-l) 
mi—m2 giQ:(ni—112-1-1) 


1 , 

1 , 

1 , 

1 . 


These relations can be rewritten in the following way: 


mi = m 2 + 1, a{ni — 112 ) = 2Trk, k G Z; (16) 

m 2 = mi -|- 1, o;(ni — 112 ) = 27rk, k G Z; (17) 

mi = m 2 , Q:(ni — n2 — 1) = 2Trk, k GZ\ (18) 

mi = m 2 , Q:(ni — n2 -|- 1) = 2'Kk, k G Z. (19) 

From these relations for integers vectors (mi, m 2 , ni, 712 ) such that |m|-|-|n| = 
mi -|-m 2 -|-rei -|-n 2 = 2, 3 we derive that if |m| -|- |n| = 2, then resonance rela¬ 
tions (16) and 0 are absent, resonance relations (|18[) and (|19[) are the same 
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as for the case of 2-dimensional elliptic point: a = 27r/3. Thus, according 
to our assumptions (that p is l-elliptic hxed point of generic type for f\w‘^) 
resonant monomials of second order can be removed. If \m\ + \n\ = 3, then 
for resonance relations (16) we get resonant monomials x^y, corresponding 
to 2-dimensional saddle point of symplectic diffeomorphism, and x{u^ +v‘^). 
For relations (17) we get xy'^ and y{u^ +v‘^). Under an assumption that 
strong resonances are absent in the system, relations (18) and (19) show res¬ 
onant monomials xyu, xyv, u{v? +v^) and v{v?‘ cannot be eliminated. 

The last two present in the normal form for a diffeomorphism in a neighbor¬ 
hood of an elliptic point. Taking into account that the transformation to the 
normal form should be symplectic we get (15). ■ 


7.3 Invariant foliations and their smoothness 

In this subsection we verify the conditions from |15( flH] which guarantee the 
existence and smoothness of stable and unstable invariant foliations within 
manifolds VF“, VF™, respectively. In particular, these conditions imply the 
existence of stable and unstable smooth invariant manifolds for KAM-curves 
on the center manifold Homoclinic orbits to KAM-curves belongs to in¬ 
tersection of these manifolds. Note that this fact does not follow immediately 
from the Hirsch-Pugh-Shub theorem | l21j and we use the theory developed by 
Fenichel dmiB]. Let us recall the dehnition of weakly overflowing invariant 
set according to [15] : 

Definition 1 (N. Fenichel, 1974) Let U and V be open subsets of some 
-manifold Mi, 1 < I < oo, and let F -.U ^ V be a -diffeomorphism. A 
set A G U is called weakly overflowing invariant {under F) if A G F{A). 

Let TF : TU —?■ TV be the map induced by F on tangent spaces. A sub¬ 
bundle E G TMi|^ is called weakly overflowing invariant if E G TF{E). 

Let us choose any invariant KAM-curve on in a sufficiently small 
neighborhood of p. Then the closure of subset in bounded by this KAM- 
curve is weakly overflowing invariant set under diffeomorphism /. Denote 
this compact set as A. Here we assume Mi = and consider f\j^^- 

To apply expanding family theorems one need to show that there exists 
weakly overflowing invariant sub-bundle E G TMi\^. It will be proved 
using contraction mapping principle. Recall that locally near p manifolds 
^rcs straightened (i.e. a: = 0 on LF™ and y = 0 on 1F“) and 
{y,u,v) are coordinates on it. Then the restriction / on fF'^“ has following 
form: 

= h~^yn + h{yn, Un,Vn)yn 
= Un COS a - Vn sin a-\-gi{yn,Un,Vn) 

= Un sin a-\-Vn cos a-\-g2{yn,Un,Vn), 


Un+l 

'^n+1 

^n+1 
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where functions gi,g 2 are of second order in (y, u,v), h is first order function. 
Now let us change variables {yn,Un,Vn) —^ {yn,Un,Vn)- 



In new coordinates the diffeomorphism will have the form (we keep old no¬ 
tations for variables): 


Un+l 

— M Un “1“ hijjrtt ^n)?/n 



— ti/x + Q\ {yw) '^n) 

( 20 ) 

V-n+l 

— Q2{yn')'^n')'^n) •) 



At any point m £ A differential Df has the following representation (recall 
A C is given as y = 0): 


Dfm 


^ fj, ^ + h{m) 0 0 

giy{m) l + giSm) yi„(m) 

V 92y{m) g2ui'm) l + g2A'm) 


Since we work in one coordinate chart {y,u,v), we will denote (y,Xi)A 2 ) 
as coordinates in the tangent space to a point m. Now consider any orbit 
(..., TUn, run+i,...) of m = mo, which belongs to A (that is to W^). 

At each point of this orbit choose in Tm^Mi a straight line through the origin 
in the tangent space being transversal to plane g = 0. Such straight line can 
be given parametrically: {g,PnV,QnV), y G M, functions pn and qn smoothly 
depend on m in A. Differential Dfm transforms this line to another line in 
Tf{m)Mi: 


V 

Pn+l 

Qn+l 


(p ^ + h{mn))g, 

9ly (mn) (1 gl„ {mn))pn + Pp {mn)qn 
+ h{mn) 

g2y{mn) +g2„(»T-n)pn -F (1 + g2„ (mn))gn 

+ h{mn) 


( 21 ) 


R.h.s. of the second and third relations define an operator in the Banach 
space of uniformly bounded sequences C = {{PniQn)} with norm ||C|| = 
sup(|pn| ) kn|)- Indeed, operator will transforms a uniformly bounded se- 

nGZ 

quence C, to the bounded one, as all derivatives [pify, {PiYui ( 1 / 1 ) 1 ;) ^ = 1)2, 
calculated at are small (functions pi, i = 1,2, are of second order at 
zero): 


1 ^ I ^ 9iy{rnn) +\l+gu{'mn)\\\C\\ + \9u{'mn)\\\C\\ 
- \p-^ + h{mn)\ 

< \g 2 y{mn)\ + \92y{mn)\ IICII + |1 + 92Amn)\ IICII 

“ \p~^ + h{mn) 
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To prove the operator is contracting, consider following inequalities: 


|Pn+l 




pI+i 


qI+i 


|1 + |Pn -Pnl + |gi.("^n)| |gn " Qn\ 

|/i“^ + h{mn)\ 

^ (|1 + gu(mn)| + |gi„(mn)|) ||C^ 

“ \^~^ + h{mn)\ 

g2„(mn)| |Pn -Pn| + |1 +g2„(mn)| - g^| 

||U“^ + /i(mn)| 

^ (|g2„(mn)| + |1 +g2.(m„)|) ||C^ 

“ \fi~^ + h{mn)\ 




Quantities such as 

|1 + gu("^n)| + |gl„(mn)| 

|/i"^ + h{mn)\ 

are less than 1 uniformly in n, if we are working in a neighborhood of p 
small enough, so that the operator is contracting. According to contraction 
mapping principle there exists a unique hxed point of the operator, namely 
some sequence {(pjj, Qn)}- This sequence (as the point in the related Banach 
space) depends continuously in m G A. The straight lines corresponding 
to this sequence, as m varies along A, form weakly overflowing invariant 
sub-bundle E C rMi|^. 

Next we choose a vector bundle N C TMi]^, complementary to E, that 
is TMi\^ = £' © A^. We set N = TA. According to [E] for m G A, any 
Vo ^ E and Wo G N let 


v_k = Df ^{m)vo, 
w-k = Df-^{m)wo, 


where is projection to N (note that in our case N is invariant under Df, 
so one can just let W-k = Df~^{m)wo)- Let us also dehne two numbers 

a*{m) = inf ||r;_fc|/a^ —>■ 0 as A: —)• oo, V fo £ Ll} , 

p*{m) = inf / |rc_fc|—)• 0 as A: —>■ oo, V fo £ LI, rco G A") . 

p>0 ^ ^ 

The number a*{m) is an asymptotic measure of the growth of vectors in E 
under the action of Df~^, and p*{m) is an asymptotic measure of the ratio 
of the growth of vectors in E to the growth of vectors in N under the action 
of Df-K 

Recall one more dehnition and formulate expanding family theorem [15] 
for reader’s convenience. 
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Definition 2 (N. Fenichel, 1974) The pair (A, E) is called an invariant 
set with expanding structure for f, if A is compact and weakly overflowing 
invariant, E is weakly overflowing invariant, and a*{m) < 1, p*(m) < 1 for 
all m € A. 


Theorem 4 (Expanding Family Theorem, N. Fenichel, 1974) Let Mi 

be -manifold, 1 < I < oo, and let E : Mi —)• Mi be a -diffeomorphism. 
Let (A, E) be an invariant set with expanding structure. Then there is a 
family of -manifolds W^{m), m £ A, invariant in the sense that 

F-^{W^{m)) = W^{E-^{m)). 

The manifold W^{m) is -diffeomorphic to the fiber Em, and is tangent to 
Em at m. 


Let us show that in our case a*{m) < 1, p*{m) < 1 for any m C A. Take 
any vq £ E, it has the coordinate representation: vq = (??o,Po^O) ^o%)- 
tor V-k will have the representation: V-k = {V-k^P-kh-k: Q-kV-k)- Taking 
into account first equality from (21) one gets: 


P-i = 
V-2 = 

P-k = 




1 + ph{m-i) ’ 

1 + ph{m- 2 ) (1 + ph{m-i)){l + ph{m- 2 )) ’ 






(1 + ph{m-i ))... (1 + ph{m_k))' 

Now consider the ratio 

\p-k\ + + 


T-k 


a 


a 


1(1 + ph{m_i ))... (1 + ph{m_k))\ 


Quantity |7/o| y^l + {p*_]f)‘^ + (<?* is bounded. Function h is of the first 
order, m-n lies in small neighborhood of fixed point p. Let us define = 
sup|/i(mn,)|. This value is of order of size (radius) of neighborhood and hence 

is small enough. The following estimates are valid: 

v'l + (Pl+ (oil)" |„_^| /l<)olv'l + (pli)=+ («!»)= 


a'‘(l+^lS,r 


\V-k\ 

< I— Ll < 


afll-pSiY 


( 22 ) 

Quantity in the r. h. s. of inequality (22) tends to zero as A; —)• oo if p/a{l — 
p6i) < 1, that is a > /x/(l — p6i). On the other hand, quantity in the 1. h. s. 
nds to zero 

a*{m(i) £ ■:r—^ —^ < 1- 


of inequality (22) tends to zero as /c —>• oo if a > /i/(l + pdi). Thus, we get: 

p p 


1 + p6i ’ 1 — p6i 
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The inverse map for Dfm„ has following representation: 


Df^l = 


h 

1 + ^h{m„ 


\ 


0 

1 + hinir, 

kirrin) 



(23) 


where dots, li, I 2 , h and are of at least first order fnnctions. Take any 
wq G N, it has the form: wq = (0, cq, do)) and let ns consider k-th iteration of 
rco nnder DF~^: W-k = From (23) one gets that coordinates 

of W-k change as follows: 


C-k — + h{nT'-k+i))c-k+i + h{'m'-k+i)d-k+i 

d-k = h{rn-k+i)c-k+i + {'^ + U{'m-k+i))d-k+i- 


Denote 62 = supsup|/j(m,i)|, i = 1,4. This qnantity is also small enongh 

i n<^'L 

and of the order of the size of neighborhood. Next estimates are valid: 

\C-k\ < (1 + d 2 )|c_A;+l| + (52|d-A;+l| = |c_fc+i| + h 2 (|c_fc+l| + |d_A:+l|) 
\d-k\ < (52|c_fc+i| + (1 + (52)|d_fc+i| = |d-fc+i| + h2(|c_fc+i| + |d_fc+i|), 


and, so: 


C—k\ + \d—k 


< |c-fc+i| + |d_fe+i| + 262 {\c-k+i\ + |d_fc+i|) 

= (|C-A;+l| + |d-fc+l|)(l + 2^2) 

^ (|C-A;+2| + |d_fc+2|)(l + 2 ^ 2 )^ < ■ • • 

— (|co| + |do|)(l + 2(52)^. 


Then one gets: 


V^|c_A:P + |d-fc|2 < ^2(max{|c_fc|, \d_k\}f < V2 max{|c_/i;|, |(i-/c|} 

— (|c-fc| + |d_fc|) < \/2(|co| + |do|)(l + 262)^■ 

On the other side: 


\C-k\ > (1 — d 2 )|c_fc+l| — (52|d-A;+l| = |c_fc+l| — (52(|C-fc+l| + |d-fc+l|) 
\d-k\ > (1 — (52)|d-A;+l| — d2|c_fc+l| = |d-fc+l| — h2(|c-fc+l| + |d-A:+l|), 


and, conseqnently. 


C—fc| + \d—k 


> |c-fc+i| + |d_fc_|_i| — 2h2(|c_fc+i| + |d_fc+i|) 

= (|c-fc+i| + |d-fc+i|)(l — 2 ^ 2 ) 

> (ic-A;+2i + id_fc+2i)(l — 2 ^ 2 )^ > ■ • • 

> (|co| + |do|)(l-2<52)^ 


The following ineqnalities are valid: 

(|co| + |do|)(l - 2 ^ 2 )^ < \c-k\ + \d-k\ < 2max{|c_A:|, |d_A:|} , 
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so 


max{|c_fc|, \d_k\} > ^ (1 - 2 ^ 2 )^ 

and 

\/|c_A:p + |d-fcp > \J (max{|c_fc|, \d-k\}f = max{|c_fc|, |d-fc|} > ^ ( 1 + 2 ^ 2 )^- 

Now let us evaluate 

|„_t| _ M^lml^i + iPUf + iiUr- _ 

\w-k\ P^ 1(1 + p/i(m_i))... (1 + p/i(m_fc))| i/|c_fc|2 + |(i_fcP 
Next estimates are valid: 


I„_t| ^ p l^ol yi + (Pi*)" + (91.)“ 

r/;-A:|p" - p'=(l + p<5i)'=v/2(|co| + |do|)(l + 2<52)" 

|«_,| ^ 2Pkol\/l + {pU)^ + 

u._t|p‘ - /(l-Mii)''(|col + l4l)(l-2&)'‘' 


The expression in the r. h. s. of the first inequality in ( |24[ ) tends to zero as 
k tends to 00 if p > /i/(l + /rJi)(l + 282 ), while expression in the r. h. s. of 
the second inequality in (24) tends to zero as k tends toooifp>/u/(l — 
/i5i)(l — 262 )- Thus, we get: 


p*{mo) G 


_P_ _P_ 

(l + /r<5i)(l + 2<52)’ (l-pJi)(l-2<52) 


< 1 . 


Then expanding family theorem holds and for each point m G A there exists 
1-dimensional manifold (a curve) in Mi being tangent to corresponding layer 
in E. The collection of these curves is invariant under In particular, 

collecting these manifolds for points of an invariant KAM-curve defines its 
unstable manifold. 

Now we want to have smoothness properties for the expanding foliation 
obtained. Let us apply smoothness theorem for invariant sets with expanding 
structure to prove that these manifolds smoothly depend on mo |16) . For 
this purpose we define the quantity t* according to |16) : 


T*{mo) = inf {r : [|t>_fc| / \w-k\f |C-A:| 0 as A: 00 , 

y Vo ^ E, Wo G A^, ^0 G Tra^K = A^} . 

Theorem 5 (Smooth Invariant Bundle Theorem, N. Fenichel, 1977) 

Let U and V be open subsets of a -manifold Mi, and let E : U ^ V be 
a -diffeomorphism, 2 < I < 00 . Let A. be a compact, properly embedded, 
C' -manifold with boundary, overflowing invariant under F. Let (A, F) be an 
invariant set with expanding structure. If 1 < f < I — 1, and T*(m) < Ijl' 
for all m £ A, then E is a -smooth vector bundle. 
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Taking into account estimates found before, one gets for diffeomorphism 
(20) and any vector = {0,Co,do) G N, = {0,C-k, d-k), following 
inequality to be valid: 


\v-k\ 

\w-k\ 






1(1 + ... (1 + ^ih{m-k))\ i/lc-fcp + 


c_fcp + |d-fcp 


^rk^T 

/A 


< 


{1 - - 262 ) 


\Tk 


^/2(|co| + |do|)(l + 2,52)^ (25) 


here Ci is constant which can be easily calculated. The r. h. s. of (25) tends 
to zero as /c —)• 00 if 


r > Ti = 


ln(l + 262 ) 


> 0 . 


ln[(l - /xhi)(l - 282 )] - In/i 
Quantity ri is small enough of order <52- On the other hand, 

I Co I + I do I 


\v-k\ 

\w-k\ 


n?-A;| > 




(1 + (1 + 262) 


rk 


(1 - 2 ^ 2 )^ 


(26) 


The r. h. s. of (26) tends to zero as /c —)> 00 if 

ln(l - 252) 


r > T 2 = 


ln[(l + + 282 )] - In^ 


, r2 < 0. 


Therefore, r*(mo) G [t 2 ,ti], so vector bundle E is smooth, where r' < 
^/ti, r' < r — 2 . 

Existence of stable manifold and its smoothness can be proved in 

a similar way. 
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